On Out-of-Sample Statistics for Financial Time-Series

This paper studies an out-of-sample statistic for time-series prediction that is analogous to the widely used R in-sample statistic. We propose and study methods to estimate the variance of this out-of-sample statistic. We suggest that the out-of-sample statistic is more robust to distributional and asymptotic assumptions behind many tests for insample statistics. Furthermore we argue that it may be more important in some cases to choose a model that generalizes as well as possible rather than choose the parameters that are closest to the true parameters. Comparative experiments are performed on artificial data as well as on a financial time-series (daily and monthly returns of the TSE300 index). The experiments are performed for varying prediction horizons and we study the relation between predictibility (out-of-sample R), variability of the outof-sample R statistic, and the prediction horizon. In particular, we find that very different conclusions would be obtained when testing against the null hypothesis of no dependency rather than testing against the null hypothesis that the proposed model does not generalize better than a naive forecast.